System and Methods for Customizing Neural Networks

ABSTRACT

Provided herein is a system including, in some embodiments, one or more servers and one or more database servers configured to receive user-specific target information from a client application for training a neural network on a neuromorphic integrated circuit. The one or more database servers are configured to merge the user-specific target information with existing target information to form merged target information in the one or more databases. The system further includes a training set builder and a trainer. The training set builder is configured to build a training set for training a software-based version of the neural network from the merged target information. The trainer is configured to train the software-based version of the neural network with the training set to determine a set of synaptic weights for the neural network on the neuromorphic integrated circuit.

PRIORITY

This application is a continuation of U.S. patent application Ser. No.16/164,671, filed Oct. 18, 2018, now issued U.S. Pat. No. 11,373,091,which claims the benefit of priority to U.S. Provisional PatentApplication No. 62/574,650, filed Oct. 19, 2017, which are incorporatedby reference in their entireties herein.

FIELD

Embodiments of the disclosure related to the field of neuromorphiccomputing. More specifically, embodiments of the disclosure relate tosystems and methods for customizing a neural network of a neuromorphicintegrated circuit.

BACKGROUND

In computer science, machine learning is a discipline of artificialintelligence (“AI”) directed to providing machines the ability to learnfrom data and make predictions on such data. One branch of machinelearning includes, for example, deep learning, which utilizes deep, ormultilayered, neural networks to learn from data and make predictions onsuch data. Machine learning is becoming commonplace for numerousapplications including bioinformatics, computer vision, video games,marketing, medical diagnostics, online search engines, and othermachine-learning applications; however, conventional central processingunits (“CPUs”) that process instructions based on “clocked time” are notable to process the data required of machine-learning applications whilekeeping power consumption low. There are research efforts to developdirect hardware implementations of deep neural networks that aim to keeppower consumption low, but existing research efforts still fall short ofproviding the requisite processing power for machine-learningapplications—particularly custom processing capabilities for newapplications and new situations in existing applications—while keepingpower consumption low enough for everyday use such as in stand-alonebattery-powered devices. Disclosed herein are systems and methods forcustomizing neural networks with custom processing capabilities for newapplications and new situations in existing applications.

SUMMARY

Provided herein is a system for customizing neural networks including,in some embodiments, one or more servers and one or more databaseservers. The one or more servers are configured to receive user-specifictarget information from a client application for training a neuralnetwork on a neuromorphic integrated circuit. The one or more databaseservers are configured to receive the user-specific target informationfrom the one or more servers and merge the user-specific targetinformation in one or more databases. The one or more databases includeexisting target information. The merge of the user-specific targetinformation forms merged target information in the one or moredatabases. The system further includes a training set builder and atrainer. The training set builder is configured to build a training setfor training a software-based version of the neural network from themerged target information. The trainer is configured to train thesoftware-based version of the neural network with the training set todetermine a set of synaptic weights for the neural network on theneuromorphic integrated circuit.

In some embodiments, the training set builder is further configured tobuild the training set to use existing training for the existing targetinformation.

In some embodiments, the trainer is further configured to update thesynaptic weights for the software-based neural network for previouslylearned existing target information in view of newly learneduser-specific target information.

In some embodiments, the system further includes a file builderconfigured to build a file of the set of synaptic weights for updatingfirmware of the neuromorphic integrated circuit including the neuralnetwork. The one or more servers are further configured to provide thefile to the client application for updating the firmware of theneuromorphic integrated circuit with the set of synaptic weights for theneural network.

Provided herein is a system for customizing neural networks including,in some embodiments, one or more servers and one or more databaseservers. The one or more servers are configured to receive user-specifictarget information from a client application for training a neuralnetwork on a neuromorphic integrated circuit. The one or more databaseservers are configured to receive the user-specific target informationfrom the one or more servers and merge the user-specific targetinformation in one or more databases. The one or more databases includeexisting target information. The merge of the user-specific targetinformation forms merged target information in the one or moredatabases. The system further includes a training set builder and atrainer. The training set builder is configured to build a training setfor training a software-based version of the neural network from themerged target information. The trainer is configured to train thesoftware-based version of the neural network with the training set todetermine a set of synaptic weights for the neural network on theneuromorphic integrated circuit. The system further includes a filebuilder. The file builder is configured to build a file of the set ofsynaptic weights for updating firmware of the neuromorphic integratedcircuit including the neural network.

In some embodiments, the training set builder is further configured tobuild the training set to use existing training for the existing targetinformation. The trainer is further configured to update the synapticweights for the software-based neural network for previously learnedexisting target information in view of newly learned user-specifictarget information. The one or more servers are further configured toprovide the file to the client application for updating the firmware ofthe neuromorphic integrated circuit with the set of synaptic weights forthe neural network.

Also provided herein is a method for customizing neural networksincluding, in some embodiments, receiving user-specific targetinformation for training a neural network on a neuromorphic integratedcircuit; merging the user-specific target information with existingtarget information to form merged target information; building atraining set for training the neural network from the merged targetinformation; and training the neural network to determine a set ofsynaptic weights for the neural network. Receiving the user-specifictarget information includes receiving the user-specific targetinformation from a client application by one or more servers. Receivingthe user-specific target information further includes receiving theuser-specific target information from the one or more servers by one ormore database servers. Merging the user-specific target information withexisting target information includes merging the user-specific targetinformation in one or more databases including the existing targetinformation to form the merged target information in the one or moredatabases. Building the training set for training the neural networkincludes building the training set with a training set builder, whereinthe training set is for training a software-based version of the neuralnetwork from the merged target information. Training the neural networkincludes training the software-based version of the neural network witha trainer on the training set to determine the set of synaptic weightsfor the neural network on the neuromorphic integrated circuit.

In some embodiments, the method further includes labeling the targetinformation in the one or more databases before merging theuser-specific target information in the one or more databases. Thelabeling includes labeling keywords for keyword spotting.

In some embodiments, building the training set includes building thetraining set configured to use existing training for the existing targetinformation.

In some embodiments, the method further includes updating, with thetrainer, the synaptic weights for the software-based neural network forpreviously learned existing target information in view of newly learneduser-specific target information. The updating includes training thesoftware-based neural network on information other than the existingtarget information already learned.

In some embodiments, the method further includes building, with a filebuilder, a file of the set of synaptic weights for updating firmware ofthe neuromorphic integrated circuit including the neural network.

In some embodiments, the method further includes providing the file tothe client application, thereby allowing the firmware of theneuromorphic integrated circuit to be updated with the set of synapticweights for the neural network.

In some embodiments, the method further includes labeling the targetinformation in the one or more databases before merging theuser-specific target information in the one or more databases, whereinthe labeling includes labeling keywords for keyword spotting; updating,with the trainer, the synaptic weights for the software-based neuralnetwork for previously learned existing target information in view ofnewly learned user-specific target information, wherein the updatingincludes training the software-based neural network on information otherthan the existing target information already learned; and building, witha file builder, a file of the set of synaptic weights for updatingfirmware of the neuromorphic integrated circuit including the neuralnetwork; providing the file to the client application, thereby allowingthe firmware of the neuromorphic integrated circuit to be updated withthe set of synaptic weights for the neural network. Building thetraining set includes configuring the training set to use existingtraining for the existing target information.

DRAWINGS

Embodiments of this disclosure are illustrated by way of example and notby way of limitation in the figures of the accompanying drawings, inwhich like references indicate similar elements and in which:

FIG. 1 illustrates a system for designing and updating neuromorphicintegrated circuits (“ICs”) in accordance with some embodiments.

FIG. 2 illustrates an analog multiplier array in accordance with someembodiments.

FIG. 3 illustrates an analog multiplier array in accordance with someembodiments.

FIG. 4 illustrates a bias-free, two-quadrant multiplier of an analogmultiplier array in accordance with some embodiments.

FIG. 5 illustrates a system for customizing neural networks onneuromorphic ICs in accordance with some embodiments.

FIG. 6 illustrates a user on a client machine interfacing with a webapplication coupled to the system of FIG. 5 in accordance with someembodiments.

FIG. 7A illustrates a method for customizing neural networks onneuromorphic ICs in accordance with some embodiments.

FIG. 7B illustrates a method for merging user-specific targetinformation in a database in accordance with some embodiments.

FIG. 7C illustrates a method for building a training set in accordancewith some embodiments.

FIG. 7D illustrates a method for updating firmware of a neuromorphic ICwith new synaptic weights in accordance with some embodiments.

FIG. 8 illustrates a method for customizing neural networks onneuromorphic ICs in accordance with some embodiments.

DESCRIPTION

In the following description, certain terminology is used to describefeatures of the invention. For example, in certain situations, the term“logic” can be representative of hardware, firmware, or software that isconfigured to perform one or more functions. As hardware, logic caninclude circuitry having data processing or storage functionality.Examples of such circuitry can include, but are not limited orrestricted to a microprocessor, one or more processor cores, aprogrammable gate array, a microcontroller, a controller, an applicationspecific integrated circuit, wireless receiver, transmitter ortransceiver circuitry, semiconductor memory, or combinatorial logic.

The term “process” can include an instance of a computer program (e.g.,a collection of instructions, also referred to herein as anapplication). In one embodiment, the process can be included of one ormore threads executing concurrently (e.g., each thread can be executingthe same or a different instruction concurrently).

The term “processing” can include executing of a binary or script, orlaunching an application in which an object is processed, whereinlaunching should be interpreted as placing the application in an openstate and, in some implementations, performing simulations of actionstypical of human interactions with the application.

The term “object” generally refers to a collection of data, whether intransit (e.g., over a network) or at rest (e.g., stored), often having alogical structure or organization that enables it to be categorized ortyped. Herein, the terms “binary file” and “binary” will be usedinterchangeably.

The term “file” is used in a broad sense to refer to a set or collectionof data, information or other content used with a computer program. Afile can be accessed, opened, stored, manipulated or otherwise processedas a single entity, object or unit. A file can contain other files, andcan contain related or unrelated contents or no contents at all. A filecan also have a logical format or be part of a file system having alogical structure or organization of plural files. Files can have aname, sometimes called simply the “filename,” and often appendedproperties or other metadata. There are many types of files, such asdata files, text files, program files, and directory files. A file canbe generated by a user of a computing device or generated by thecomputing device. Access or operations on a file can be mediated by oneor more applications or the operating system of a computing device. Afilesystem can be configured to organize the files of the computingdevice of a storage device. The filesystem can enable tracking of filesand enable access of those files. A filesystem can also enableoperations on a file. In some embodiments the operations on the file caninclude file creation, file modification, file opening, file reading,file writing, file closing, and file deletion.

Lastly, the terms “or” and “and/or” as used herein are to be interpretedas inclusive or meaning any one or any combination. Therefore, “A, B orC” or “A, B and/or C” mean “any of the following: A; B; C; A and B; Aand C; B and C; A, B and C.” An exception to this definition will occuronly when a combination of elements, functions, steps or acts are insome way inherently mutually exclusive.

Again, disclosed herein are systems and methods for customizing neuralnetworks with custom processing capabilities for new applications andnew situations in existing applications.

Referring now to FIG. 1, a system 100 is illustrated for designing andupdating neuromorphic ICs in accordance with some embodiments. As shown,the system 100 can include a simulator 110, a neuromorphic synthesizer120, and a cloud 130 configured for designing and updating neuromorphicICs such as neuromorphic IC 102. As further shown, designing andupdating neuromorphic ICs can include creating a machine learningarchitecture with the simulator 110 based on a particular problem. Theneuromorphic synthesizer 120 can subsequently transform the machinelearning architecture into a netlist directed to the electroniccomponents of the neuromorphic IC 102 and the nodes to which theelectronic components are connected. In addition, the neuromorphicsynthesizer 120 can transform the machine learning architecture into agraphic database system (“GDS”) file detailing the IC layout for theneuromorphic IC 102. From the netlist and the GDS file for theneuromorphic IC 102, the neuromorphic IC 102, itself, can be fabricatedin accordance with current IC fabrication technology. Once theneuromorphic IC 102 is fabricated, it can be deployed to work on theparticular problem for which it was designed. While the initiallyfabricated neuromorphic IC 102 can include an initial firmware withcustom synaptic weights between the nodes, the initial firmware can beupdated as needed by the cloud 130 to adjust the weights. Being as thecloud 130 is configured to update the firmware of the neuromorphic IC102, the cloud 130 is not needed for everyday use of the neuromorphic IC102.

Neuromorphic ICs such as the neuromorphic IC 102 can be up to 100× ormore energy efficient than graphics processing unit (“GPU”) solutionsand up to 280× or more energy efficient than digital CMOS solutions withaccuracies meeting or exceeding comparable software solutions. Thismakes such neuromorphic ICs suitable for battery powered applications.

Neuromorphic ICs such as the neuromorphic IC 102 can be configured forapplication specific standard products (“ASSPs”) including, but notlimited to, keyword spotting, speaker identification, one or more audiofilters, speech enhancement, gesture recognition, image recognition,video object classification and segmentation, or autonomous vehiclesincluding drones. For example, if the particular problem is one ofkeyword spotting, the simulator 110 can create a machine learningarchitecture with respect to one or more aspects of keyword spotting.The neuromorphic synthesizer 120 can subsequently transform the machinelearning architecture into a netlist and a GDS file corresponding to aneuromorphic IC for keyword spotting, which can be fabricated inaccordance with current IC fabrication technology. Once the neuromorphicIC for keyword spotting is fabricated, it can be deployed to work onkeyword spotting in, for example, a system or device.

Neuromorphic ICs such as the neuromorphic IC 102 can be deployed intoys, sensors, wearables, augmented reality (“AR”) systems or devices,virtual reality (“VR”) systems or devices, mobile systems or devices,appliances, Internet of things (“IoT”) devices, or hearing devices orsystems.

Referring now to FIG. 2, an analog multiplier array 200 is illustratedin accordance with some embodiments. Such an analog multiplier array canbe based on a digital NOR flash array in that a core of the analogmultiplier array can be similar to a core of the digital NOR flasharray. That said, at least select and read-out circuitry of the analogmultiplier array are different than a digital NOR array. For example,output current is routed as an analog signal to a next layer rather thanover bit lines going to a sense-amp/comparator to be converted to a bit.Word-line analogs are driven by analog input signals rather than adigital address decoder. Furthermore, the analog multiplier array 200can be used in neuromorphic ICs such as the neuromorphic IC 102. Forexample, a neural network can be disposed in the analog multiplier array200 in a memory sector of a neuromorphic IC. However, the neural networkis not limited to the foregoing. The neural network can further includea digital layer in addition to analog layers of the neural networkdisposed in the analog multiplier array 200.

Since the analog multiplier array 200 is an analog circuit, input andoutput currents can vary in a continuous range instead of simply on oroff. This is useful for storing synaptic weights (aka weights orcoefficients) of the neural network as opposed to digital bits. Inoperation, the weights are multiplied by input currents to provideoutput currents that are combined to arrive at a decision of the neuralnetwork.

The analog multiplier array 200 can utilize standard programming anderase circuitry to generate tunneling and erase voltages.

Referring now to FIG. 3, an analog multiplier array 300 is illustratedin accordance with some embodiments. The analog multiplier array 300 canuse two transistors (e.g., a positive metal-oxide-semiconductorfield-effect transistor [“MOSFET”] and a negative MOSFET) to perform atwo-quadrant multiplication of a signed weight (e.g., a positive weightor a negative weight) and a non-negative input current. If an inputcurrent is multiplied by a positive or negative weight, the product oroutput current can respectively be either positive or negative. Apositively weighted product can be stored in a first column (e.g.,column corresponding to I_(Out0+) in the analog multiplier array 300),and a negatively weighted product can be stored in a second column(e.g., column corresponding to I_(Out0−) in the analog multiplier array300). The foregoing positively and negatively weighted products oroutput signals can be taken as a differential current to provide usefulinformation for making a decision.

Because each output current from the positive or negative transistor iswired to ground and proportional to the product of the input current andthe positive or negative weight, respectively, the power consumption ofthe positive or negative transistor is near zero when the input currentsor weights are at or near zero. That is, if the input signals are ‘0,’or if the weights are ‘0,’ then no power will be consumed by thecorresponding transistors of the analog multiplier array 300. This issignificant because in many neural networks, often a large fraction ofthe weights are ‘0,’ especially after training. Therefore, energy issaved when there is nothing to do or going on. This is unlikedifferential pair-based multipliers, which consume a constant current(e.g., by means of a tail bias current) regardless of the input signal.

Referring now to FIG. 4, a bias-free, two-quadrant multiplier 400 of ananalog multiplier array such as the analog multiplier array 300 isillustrated in accordance with some embodiments. Because each outputcurrent from the positive transistor (e.g., M1 of the two-quadrantmultiplier 400) or the negative transistor (e.g., M2 of the two-quadrantmultiplier 400) is wired to ground and proportional to the product ofthe input current and the positive or negative weight, respectively, thepower consumption of the positive or negative transistor is at or nearzero when the input currents or weights are at or near zero. That is, ifthe input signals are ‘0,’ or if the weights are ‘0,’ then no power willbe consumed by the corresponding transistors of the analog multiplierarray 300. This is significant because in many neural networks, often alarge fraction of the values or the weights are ‘0,’ especially aftertraining. Therefore, energy is saved when there is nothing to do orgoing on. This is unlike differential pair-based multipliers, whichconsume a constant current (e.g., by means of a tail bias current)regardless of the input signal.

When programming a two-quadrant multiplier such as the bias-free,two-quadrant multiplier 400, each programmable cell can be erased (e.g.,the cell including transistor M1 and the cell including transistor M2)thereof to set the cells to one extreme weight before setting each ofthe cells to its target weight. Extending this to a full array such asthe analog multiplier array 300, all of the programmable cells in thefull array can be set to one extreme weight before setting each of thecells to its target weight. When setting the cells to their desiredweights, a problem of overshoot exists if one or more of the cells isset with a higher weight than targeted. That is all of the cells in thefull array must be reset to the one extreme weight before resetting thecells to their target weights. However, the differential structure ofeach of the bias-free, two-quadrant multipliers of the analog multiplierarrays provided herein allows for compensating such overshoot byprogramming, thereby providing a distinct advantage over currenttechnology by obviating the time-consuming process of erasing andresetting all of the cells in an array.

In an example of compensating for overshoot by programming, v_(i−) andv_(i+) of the two-quadrant multiplier 400 can be erased to set the cellsto one extreme weight. After erasing the cells, if v_(i−) is programmedwith too large a weight, v_(i+) can be programmed with a larger weightthan initially targeted to compensate for the weight of v_(i−) andachieve the initially targeted effect. Therefore, the differentialstructure can be exploited to compensate for programming overshootwithout having to erase any one or more cells and start over.

Referring now to FIG. 5, a system 500 for customizing neural networks onneuromorphic ICs is illustrated in accordance with some embodiments.FIG. 6 illustrates a user on a client machine 600 interfacing with a webapplication 650 coupled to the system 500 in accordance with someembodiments. As shown in FIGS. 5 and 6 and with reference to FIG. 1, thesystem 500 includes the cloud 130, which cloud 130 includes one or moreservers such as one or more web servers 132, one or more web applicationservers 134, and one or more database servers 136, as well as one ormore databases 138. The one or more servers 134 are implemented using amicroprocessor for operation with cloud 130. The one or more web servers132 are configured to receive user-specific target information sent fromthe web application 650 for training a neural network, wherein theneural network is on a neuromorphic IC such as the neuromorphic IC 102,an analog chip, a digital signal processor (“DSP”), or some otherplatform. The one or more web servers 132 are configured to send theuser-specific target information to the one or more web applicationservers 134, which are, in turn, configured to send the user-specifictarget information to the one or more database servers 136.

While the web application 650 is primarily described herein for sendingthe user-specific target information to the system 500, a locallyinstalled application on the client machine 600 can also be used to sendthe user-specific target information over a network connection to thesystem 500 with some modifications to the foregoing. For example, thesystem 500 need not include or at least not need to utilize the one ormore web servers 132 or web application servers 134 for receiving theuser-specific target information from the locally installed applicationon the client machine 600. Instead, the system 500 can include anapplication programming interface (“API”) configured to receive theuser-specific target information for the one or more database servers136. Regardless of the manner in which the one or more database servers136 receives the user-specific target information, the one or moredatabase servers 136 are configured to merge the user-specific targetinformation in the one or more databases 138.

The one or more databases 138 are configured to include targetinformation, which can be updated over time with additional targetinformation such as the user-specific target information. As usedherein, “existing target information” includes any existing targetinformation in the one or more databases 138 prior to an instance ofmerging the existing target information with a set of user-specifictarget information to form merged target information. With this in mind,the one or more database servers 136 are configured to receive theuser-specific target information from the one or more web applicationservers 134 and merge the user-specific target information to form themerged target information in the one or more databases 138.

Target information or target signals depend upon the ASSPs for which theneuromorphic ICs are designed, which ASSPs include, but are not limitedto, keyword spotting, speaker identification, one or more audio filters,gesture recognition, image recognition, video object classification andsegmentation, or autonomous vehicles including drones. As such, targetinformation can include specific keywords for keyword spotting, specificspeakers for speaker identification, image classes for imagerecognition, and so on. For example, a neuromorphic IC designed for avoice-controlled coffee maker can be customized such that a neuralnetwork on the coffee maker's neuromorphic IC recognizes targetinformation or keywords such as “brew,” stop,” “more,” “less,” etc. Forexample, a neuromorphic IC designed for a voice-controlled airconditioner can be customized such that a neural network on the airconditioner's neuromorphic IC recognizes target information or keywordssuch as “warmer,” cooler,” “on,” “off,” etc. For example, a neuromorphicIC designed for smart home can be customized such that a neural networkon a neuromorphic IC of the smart home recognizes target information orkeywords such as “door unlock,” “door lock,” etc. For example, aneuromorphic IC designed for a smart camera can be customized such thata neural network on the smart camera's neuromorphic IC recognizes targetinformation or image classes such as humans, animals, etc. Theneuromorphic IC designed for the smart camera can be further customizedsuch that the neural network on the smart camera's neuromorphic ICrecognizes target information or acoustic signals such as gunshots,breaking glass, etc.

While not expressly shown in FIG. 5, the system 500 further includes atraining set builder, a trainer, and a file builder in some embodiments.The training set builder is configured to build a training set (e.g., auser-specific training set) for training a software-based neural networkon the system 500 from the merged target information. The training setcan be built to use existing training for existing target information soas to not waste time and resources retraining the software-based neuralnetwork on that the neural network has already learned. The trainer isconfigured to train the software-based neural network with the trainingset to determine a set of synaptic weights for the neural network. Thetrainer is further configured to update the synaptic weights for thesoftware-based neural network for previously learned existing targetinformation in view of newly learned user-specific target information.Updating the synaptic weights need not include retraining, and in thisway, the trainer, too, saves time and resources by not retraining thatwhich the software-based neural network has already learned.

The foregoing software-based neural network on the system 500 is asoftware representation of a neural network on a neuromorphic IC such asthe neuromorphic IC 102. The neuromorphic IC 102 can be the user'sneuromorphic IC such as a customer's neuromorphic IC for which theuser-specific target information was uploaded to the system 500. Assuch, the synaptic weights for an existing neuromorphic IC or a designfor a neuromorphic IC can be easily updated with custom processingcapabilities for new applications or new situations in existingapplications. The file builder of the system 500 is configured to builda file of the set of synaptic weights for updating firmware of theneuromorphic IC including the neural network as described in referenceto at least FIG. 1 and FIG. 7D.

Referring now to FIG. 7A, a method 700A for customizing neural networkson neuromorphic ICs is illustrated in accordance with some embodiments.As shown in FIG. 7A, the method 700A includes a process step 702 ofmerging user-specific target information 701A with existing targetinformation 701B from one or more databases to provide merged targetinformation in the one or more databases 704. As set forth herein, theone or more database servers 136 are configured to receive theuser-specific target information from the one or more web applicationservers 134, and the one or more database servers 136 are furtherconfigured to automatically merge the user-specific target informationwith the existing target information from the one or more databases 138to provide the merged target information in the one or more databases138. The method 700A further includes a processes step 706 includingbuilding a training set (e.g., a user-specific training set), a processstep 708 including training a software-based neural network with thetraining set, and a process step 710 including providing a set ofsynaptic weights for a neuromorphic IC. As set forth herein, thetraining set builder is configured to build the training set, and thetrainer is configured to train the software-based neural network.

Referring now to FIG. 7B, a method 700B for merging user-specific targetinformation in a database is illustrated in accordance with someembodiments. As shown, the method 700B for merging the user-specifictarget information 701A in a database such as the one or more databases138, a copy of the database, or a subset of the database includes aprocess step 712 of determining whether a specific instance of theuser-specific target information (e.g., specific keyword, specificspeaker, image, etc.) exists as existing target information in thedatabase. If the specific instance of the user-specific targetinformation exists in the database as existing target information, thespecific instance of the user-specific target information need not bemerged in the database, and the specific instance of the user-specifictarget information in the existing target information is used instead(process step 714). If the specific instance of the user-specific targetinformation does not exist in the database as existing targetinformation, the specific instance of the user-specific targetinformation is merged in the database for building a training set (e.g.,a user-specific training set) (process step 702).

While not shown in FIG. 7B, merging the user-specific target informationin the database further includes labeling the target information in thedatabase, a copy of the database, or a subset of the database forbuilding the training set (e.g., a user-specific training set). Table 1set forth below shows example labels for existing target information ina database (e.g., the one or more databases 138) including keywords forkeyword spotting. If a specific instance of the user-specific targetinformation exists in the database as existing target information, thespecific instance of the user-specific target information need not berelabeled as desired target information in the merged targetinformation. If the user-specific target information includes backgroundinformation (e.g., background noise in the context of keyword spotting)and such background information exists in the database as existingtarget information, the background information is relabeled as such inthe merged target information. Table 2 set forth below shows examplelabels for merged target information in a database (e.g., the one ormore databases 138) including keywords for keyword spotting, whereincertain instances of existing target information are relabeled as“noise” or “other.” (See Tables 1 and 2 and compare.) If a specificinstance of the user-specific target information does not exist in thedatabase as existing target information, the specific instance of theuser-specific target information is merged with the existing targetinformation and provided an appropriate label for desired targetinformation in the merged target information.

TABLE 1 Existing target information in the database. Database File Labelon_01.wav ‘on’ on_02.wav ‘on’ off_01.wav ‘off’ off_02.wav ‘off’hot_01.wav ‘hot’ hot_02.wav ‘hot’ cold_01.wav ‘cold’ cold_02.wav ‘cold’door_unlock01.wav ‘door_unlock’ door_unlock02.wav ‘door_unlock’door_lock01.wav ‘door_lock’ door_lock02.wav ‘door_lock’ subway01.wav‘noise’ or ‘other’ restaurant01.wav ‘noise’ or ‘other’

TABLE 2 Merged target information in the database. Database File Wordon_01.wav ‘noise’ on_02.wav ‘noise’ off_01.wav ‘noise’ off_02.wav‘noise’ hot_01.wav ‘noise’ hot_02.wav ‘noise’ cold_01.wav ‘noise’cold_02.wav ‘noise’ door_unlock01 .wav ‘door_unlock’ door_unlock02.wav‘door_unlock’ door_lock01.wav ‘door_lock’ door_lock02.wav ‘door_lock’subway01.wav ‘noise’ restaurant01.wav ‘noise’

The labeled target information in the merged target information (e.g.,the merged target information of Table 2) can be subsequently extractedfrom the database for automatically building the training set (e.g., auser-specific training set). In doing so, features are extracted fromthe merged target information such as time-varying frequency content inaudio recordings. A software-based neural network can be trained torecognize the desired target information (e.g., “door unlock” and “doorlock”) and the features of the desired target information whiledisregarding the background information (e.g., “noise” or “other”) andthe features of the background information. Such training includesproviding the features as input to the software-based neural network andadjusting the synaptic weights using neural network optimizationalgorithms (e.g. gradient descent in TensorFlow) such that output of theneural network is positive when desired target information isrecognized. Such training results in a plurality of synaptic weights forprogramming a hardware-based neural network on a neuromorphic IC, ananalog chip, a DSP, or some other platform, wherein the software-basedneural network and the software-based neural network correspond to eachother. The plurality of synaptic weights are mapped to physicalquantities (e.g., currents) that individual circuit elements of thehardware-based neural network embody. Once programmed, thehardware-based neural network is configured to execute on that which thecounterpart software-based neural network was trained (e.g.,classification tasks such as keyword spotting such as “door unlock” and“door lock” in a smart home environment).

Referring now to FIG. 7C, a method 700C for building a training set isillustrated in accordance with some embodiments. As shown, the method700C for building a training set (e.g., a user-specific training set)includes a process step 716 of determining whether a software-basedneural network has been trained on a specific instance of user-specifictarget information (e.g., specific keyword, specific speaker, image,etc.). If the software-based neural network has been trained on aspecific instance of the user-specific target information, the traineris configured to build the training set to use existing training for thespecific instance of the user-specific target information (process step718). Instances of the software-based neural network having been trainedon user-specific target information can occur when the user-specifictarget information already exists in the database as existing targetinformation. If the software-based neural network has not been trainedon a specific instance of the user-specific target information, thetrainer is configured to build the training set to train the neuralnetwork on the specific instance of the user-specific target information(process step 720). The 700D concludes with a process step 722 ofbuilding the training set.

Referring now to FIG. 7D, a method 700D for updating firmware of aneuromorphic IC with new synaptic weights is illustrated in accordancewith some embodiments. As shown, the method 700D includes a process step724 of building a file of synaptic weights to upload to a neuromorphicIC. The builder is configured to build the file of synaptic weights 726from output resulting from training a software-based neural network witha training set on merged target information including user-specifictarget information. The method 700D further includes a process step 728of updating firmware of a neuromorphic IC with the file of synapticweights.

FIG. 8 illustrates a method 800 for customizing neural networks onneuromorphic ICs in accordance with some embodiments. As shown, themethod 800 includes a number of process steps. In a first process step802, a user or customer enters target signals or user-specific targetinformation into an application such as the web application 650 or alocally installed application with a network connection to the system500. In a second process step 804, an application on, for example, theone or more of database servers 136 retrieves target signals or existingtarget information from the one or more databases 138. In a thirdprocess step 806, a modified database of the one or more databases 138including merged target information is built by the one or more databaseservers 136 from the user-specific target information and the existingtarget information. For each desired target signal or specific instanceof user-specific target information, the application is configured tolabel instances of the user-specific target information in the modifieddatabase appropriately. Other non-desired target signals or instances ofuser-specific target information are labelled in the modified databaseas “other” or the like. In a fourth process step 808, the modifieddatabase is used for training a software-based neural networkrepresenting a neural network on a neuromorphic IC, wherein the trainingresults in a set of synaptic weights for the neural network. In a fifthprocess step 810, the synaptic weights are translated into a file formatto upload and store on the neuromorphic IC.

In the foregoing description, the invention is described with referenceto specific exemplary embodiments thereof. It will, however, be evidentthat various modifications and changes can be made thereto withoutdeparting from the broader spirit and scope of the invention as setforth in the appended claims.

What is claimed is:
 1. A method for customizing neural networks,comprising: receiving, by one or more servers, user-specific targetinformation from a client application for training a neural network onan integrated circuit; merging, in one or more databases, theuser-specific target information with existing target information toform merged target information in the one or more databases; labelingthe user-specific target information, wherein the labeling includeslabeling keywords for keyword spotting; and extracting one or morefeatures from the labeled user-specific target information to build atraining set, wherein the extracted one or more features includestime-varying frequency content in audio recordings.
 2. The method ofclaim 1, wherein upon a determination that a first user-specific targetinformation includes background information which already exists in theone or more databases as an existing target information, the firstuser-specific target information is re-labeled as a backgroundinformation.
 3. The method of claim 1, further comprising training asoftware-based version of the neural network on the training set todetermine a set of synaptic weights for the neural network on theintegrated circuit.
 4. The method of claim 1, further comprisinglabeling the target information in the one or more databases beforemerging the user-specific target information in the one or moredatabases, wherein the labeling includes labeling keywords for keywordspotting.
 5. The method of claim 1, wherein the training set isconfigured to use existing training for the existing target information.6. The method of claim 3, further comprising updating the synapticweights for the software-based neural network for previously learnedexisting target information in view of newly learned user-specifictarget information, wherein the updating includes training thesoftware-based neural network on information other than the existingtarget information already learned.
 7. The method of claim 6, furthercomprising building, with a file builder, a file of the set of synapticweights for updating firmware of the integrated circuit including theneural network.
 8. The method of claim 7, further comprising providingthe file to the client application to allow the firmware of theintegrated circuit to be updated with the set of synaptic weights forthe neural network.
 9. The method of claim 3, wherein the software-basedneural network is configured to recognize one or more desired targetinformation and disregard background information.
 10. The method ofclaim 1, further comprising providing a first set of desired featuresassociated with desired target information and a second set of undesiredfeatures associated with undesired information as input to thesoftware-based neural network; and adjusting the synaptic weights usingone or more neural network optimization algorithms, wherein theadjusting the synaptic weights are so performed that output of theneural network is positive when desired target information isrecognized.
 11. The method of claim 10, wherein the one or more neuralnetwork optimization algorithms includes a gradient descent inTensorFlow.
 12. The method of claim 11, further comprising creating aplurality of synaptic weights for programming a hardware-based neuralnetwork on a platform.
 13. The method of claim 12, wherein the platformcomprises at least one of: a neuromorphic integrated circuit, an analogchip, and a digital signal processor.
 14. The method of claim 12,wherein the software-based neural network and the hardware-based neuralnetwork correspond to each other.
 15. The method of claim 14, whereinthe plurality of synaptic weights are mapped to physical quantities thatindividual circuit elements of the hardware-based neural network embody.16. A system for customizing neural networks, comprising: one or moredatabase servers comprising a microprocessor configured to: (i) receiveone or more user-specific target information, and (ii) merge theuser-specific target information with existing target information in oneor more databases to form merged target information in the one or moredatabases, wherein the system is configured to label the user-specifictarget information and extract one or more features from the labeleduser-specific target information and build a training set based on theextracted information, wherein the extracted features includestime-varying frequency content in audio recordings.
 17. The system ofclaim 16, wherein the system is further configured to update synapticweights for a software-based neural network for previously learnedexisting target information in view of newly learned user-specifictarget information.
 18. The system of claim 17, wherein the system isfurther configured to build a file of the synaptic weights for updatingfirmware of the integrated circuit including the neural network, and toprovide the file to a client application, associated with the system,for updating the firmware of the integrated circuit with the set ofsynaptic weights for the neural network.
 19. The system of claim 16,wherein the system is further configured to re-label a firstuser-specific target information as background information, if the firstuser-specific target information includes background information whichalready exists in the one or more database servers as an existing targetinformation.
 20. A system for customizing neural networks, comprising:one or more database servers comprising one or more microprocessorsconfigured to: receive user-specific target information from one or moreservers; merge the user-specific target information with existing targetinformation in one or more databases to form merged target information,and label the user-specific target information, wherein the labelingincludes labeling keywords for keyword spotting; a training set builderconfigured to train a software-based version of the neural network fromthe merged target information; a trainer configured to train asoftware-based version of the neural network with the training set todetermine a set of synaptic weights for the neural network on theintegrated circuit; and a file builder configured to build a file of theset of synaptic weights for updating firmware of the integrated circuitincluding the neural network.